In the data acquisition and storage industry, it is common for large amounts of data to be recorded. The amount of data stored is increasing rapidly. The capacity of computer storage, such as hard disks, is increasing accordingly every year and new technologies for allowing multiple disks to be used collectively are being developed. To keep pace, data transfer methods have been developed to move larger volumes of data, witnessed by the use of Firewire®, USB2 and gigabit Ethernet technologies to transfer data being taken up by mainstream computer manufacturers. As it becomes possible for progressively larger amounts of data to be recorded and transferred in real time, the size of the files recorded has increased to allow greater measurement precision. This creates a problem when handling the resultant data files using the kind of computers typically available in offices, e.g. for the analysis of industrial or scientific measurements, as even though personal computers have also increased in speed and capability, they cannot yet process whole recording files, now in the order of gigabytes, in real time as expected by the user.
In the real world, the handling of large amounts of data using a personal computer typically requires reading the data from the hard drive into a memory, processing the data in the memory as required and then either storing the results or discarding the processed values. When the recorded data files are larger than the 256 Mb to 512 Mb of high bandwidth solid-state memory (also known as Random Access Memory) used currently in personal computers, then the data cannot be fully cached in this solid-state memory and has to be partially, or wholly, read either directly from the hard drive or using a memory substitution file that is known in the art as a swap file, with the associated decrease in speed, before the processor can carry out any selected mathematical operations on the data.
In the future computers will undoubtedly have an increased amount of solid-state memory and more processing power, but if the size of the data files keeps increasing at the same rate as personal computer capabilities increase, then this problem will remain.
The purpose of the invention is, therefore, to aid in overcoming the above problems.